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ABSTRACT 

One  of  the  key  force  modernisation  initiatives  for  infantry  operations  is  to  enhance  situation 
awareness  (SA)  capabilities.  The  design  and  acquisition  of  technologies  that  optimise  the 
performance  of  the  combat  soldier  must  therefore  be  guided  by  the  goal  of  enhancing  and 
maintaining  SA.  Determining  the  value  of  these  technologies  requires  a  means  of  objectively 
evaluating  a  combat  soldier's  SA.  In  this  study,  the  impact  of  Intra-Section  Radios,  which  form 
part  of  the  Soldier  Combat  System  (SCS)  enhancements,  on  SA  was  evaluated.  The  Situation 
Awareness  Global  Assessment  Technique  provides  an  objective  measure  of  SA  by  comparing  the 
real  and  perceived  situation.  In  this  study,  the  Direct  Questioning  Technique  (DQT)/  an  adapted 
version  of  the  SAGAT  was  used  to  evaluate  changes  in  SA  under  baseline  and  enhanced 
conditions  during  two  loosely  scripted  exercises  involving  Australian  Air  Field  Defence  Guards. 
Results  showed  a  significant  increase  in  overall  group  SA  during  the  enhanced  patrol  vignette. 
No  significant  results  were  found  for  the  section  attack  vignette.  The  DQT  appears  to  be  a  valid 
and  viable  methodology  that  can  be  used  in  the  evaluation  of  specific  SCS  technologies. 
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Toward  a  Methodology  for  Evaluating  the  Impact  of 
Soldier  Combat  System  Technologies  on  Situation 

Awareness 


Executive  Summary 


This  report  describes  a  methodology  for  measuring  infantry  situation  awareness  (SA).  It  is 
part  of  the  development  of  a  formal  analysis  methodology  that  can  be  applied  by  agencies 
involved  in  evaluation  of  future  technologies.  In  this  study  the  impact  of  the  intrasection 
radio  (ISR)  on  SA  was  assessed.  The  ISR  has  been  identified  as  one  of  the  Soldier  Combat 
System  (SCS)  technologies  that  provides  high  pay-off  in  terms  of  enhancing  performance. 
Part  of  the  purpose  of  this  work  was  to  identify  the  strengths  and  weaknesses  of  the 
method. 

As  technology  advances,  the  enhancement  of  operator  situation  awareness  has  become  a 
major  design  goal  for  those  developing  new  technologies  to  increase  performance.  The 
ability  to  objectively  evaluate  the  impact  of  technology  on  SA  is  therefore  essential.  The 
Situation  Awareness  Global  Assessment  Technique  (SAGAT)  provides  an  objective 
measure  of  SA  through  a  comparison  between  the  real  and  perceived  situation.  The 
SAGAT  consists  of  a  pool  of  questions  that  elicit  information  from  an  operator  in  a 
simulated  environment  across  all  three  levels  of  SA,  namely  perception,  comprehension 
and  projection.  The  simulation  is  'frozen'  at  random  while  the  operator  answers  a  series  of 
randomly  selected  questions  about  the  current  situation.  These  questions  correspond  to 
the  operator's  SA  requirements. 

A  study  consisting  of  a  baseline  and  enhanced  free-play  exercise  was  conducted  to 
investigate  the  effect  of  the  ISR  on  SA.  Ten  Air  Field  Defence  Guards  participated  in  two 
vignettes,  loosely  scripted  around  the  mission  scenario  '  Exercise  Silicon  Shrike'.  In  the 
enhanced  trial,  the  contribution  of  the  ISR  to  the  SA  of  the  individual  section  members  was 
evaluated.  Two  vignettes  (Ready  Reaction  Force  (RRF)  Patrol;  Section  Attack)  appropriate 
for  a  section  activity  were  used  in  each  trial.  The  Direct  Questioning  Technique  (DQT),  a 
modified  version  of  the  SAGAT,  was  used  to  provide  an  objective  measure  of  SA.  The 
DQT  differs  from  the  original  SAGAT  technique  in  that  it  is  applied  in  a  live  exercise 
rather  than  a  computer  simulation. 

During  both  trials,  the  scenario  was  frozen  at  three  'natural  breaks',  during  which  subjects 
answered  a  randomised  subset  of  questions  about  the  current  situation,  based  on  the  SA 
requirements  of  the  soldier.  After  the  completion  of  the  exercise,  subjects'  answers  were 
evaluated  by  a  subject  matter  expert  (SME)  on  the  basis  of  what  was  actually  happening 
during  the  activities.  Questions  were  scored  based  on  a  'percentage  correct'  method.  A 
post-experiment  interview  with  the  two  Military  SME  was  also  used  to  get  a  qualitative 
account  of  the  use  of  the  ISRs  during  the  enhanced  trial. 


Results  showed  that  in  the  enhanced  RRF  Patrol  vignette,  subjects  overall  showed  a 
significant  increase  in  SA  compared  to  baseline  conditions.  This  result  seems  logical  as 
even  though  the  ISR  was  not  used  extensively  (as  indicated  by  the  post-experiment 
interview),  there  was  still  a  greater  dissemination  of  information  throughout  the  section. 
There  was  no  significant  difference  in  SA  between  the  baseline  and  enhanced  section 
attack.  Given  that  the  ISR  was  hardly  used  during  this  activity  (as  indicated  by  the  post¬ 
experiment  interview),  this  result  again  seems  logical. 

The  DQT  and  free-play  methodology  was  able  to  successfully  discriminate  between 
baseline  and  enhanced  conditions,  even  with  the  small  sample  size.  The  key  strength  of 
the  method  is  that  it  assesses  the  subjects'  SA  by  directly  asking  them  for  information 
pertaining  to  the  operational  environment.  It  is  therefore  more  objective  than  subjective 
measures  that  rely  on  self-assessment  or  expert  judgements  on  how  good  or  poor  the 
subject's  SA  is.  In  the  future,  further  validation  of  the  methodology,  consisting  of 
additional  trials  with  larger  samples  and  a  higher  number  of  probes,  is  essential.  Audio 
and  video  recordings  of  the  activities  may  be  useful  to  support  the  determination  of  the 
'ground  truth'  by  the  SMEs.  The  DQT  appears  to  be  a  valid  and  viable  methodology  that 
can  be  used  in  the  evaluation  of  future  technologies  such  as  Head  Mounted  Displays  and 
Night  Vision  Goggles,  for  acquisition. 
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1.  Introduction 


The  modem  battlespace,  especially  that  occurring  in  an  urban  area,  presents  a  complex 
and  dynamic  operational  environment.  An  increased  prevalence  in  digitisation,  joint 
warfighting  and  coalition  operations  has  emphasised  the  importance  of  situation 
awareness  (SA)  in  tactical  environments.  Within  the  combat  arms  of  the  military,  the 
infantry  is  one  of  the  most  difficult  and  demanding  professions.  Physically,  cognitively 
and  morally  challenging,  it  is  also  extremely  dangerous.  The  infantry  is  often  the  most 
exposed  of  the  combat  arms,  with  little  or  no  physical  barrier  between  its  soldiers,  the 
enemy  and  the  environment.  Operating  against  enemies  that  are  not  easily  identified, 
and  differentiating  between  friendly,  enemy  and  civilians,  is  but  one  example  of  the 
more  cognitively  challenging  aspects  of  infantry  operations  in  the  urban  environment. 

The  Land  125  Soldier  Combat  System  (SCS)  is  a  program  to  modernise  the  dismounted 
infantry  and  provide  them  with  advanced  capabilities  including  new  technologies.  The 
intra-section  radio  (ISR)  has  been  identified  as  a  technology  that  provides  high  pay-off 
in  terms  of  enhancing  performance  [1],  The  use  of  ISRs  facilitates  communication 
between  section  members  and  may  be  expected  to  enhance  their  SA.  On  the  other  hand 
SA  may  decrease  if  the  radios  are  used  inappropriately. 

Previous  research  [1-3]  evaluating  the  impact  of  technological  enhancements  on 
Australian  soldiers'  capabilities  have  used  performance-based  measures  as  the  primary 
dependent  variable. 

In  the  current  study  the  impact  of  the  ISRs  on  SA  is  assessed  using  the  Direct 
Questioning  Technique  (DQT)  [4]  based  on  the  Situation  Awareness  Global  Assessment 
Technique  (SAG AT)  [5].  It  is  part  of  the  development  of  a  formal  analysis  methodology 
that  can  be  applied  by  agencies  involved  in  evaluation  of  future  technologies.  As  such, 
part  of  the  purpose  of  the  work  is  to  identify  the  strengths  and  weaknesses  of  the 
method. 


2.  Situation  Awareness 


2.1  Definition 

The  Australian  Army  acknowledges  that  the  development  and  maintenance  of  SA  is  a 
key  component  in  both  operational  decision-making  and  effective  command  and 
control;  whilst  poor  SA  can  lead  to  catastrophic  errors  [6]. 

Although  the  phrase  'situation  awareness'  is  frequently  used  by  the  armed  forces  from 
different  nations,  there  is  no  universally  accepted  definition.  At  its  most  basic 
conception,  situation  awareness  is  generally  understood  to  mean  "knowing  what  is 
going  on  around  you"  [7].  Within  the  research  community  the  definition  of  SA  put 
forward  by  Endsley  [5]  has  been  widely  accepted  in  various  domains:  "the  perception  of 
elements  in  the  environment  within  a  volume  of  time  and  space,  the  comprehension  of 
their  meaning  and  projection  of  their  status  into  the  near  future".  This  definition  is 
useful  as  it  provides  structure  to  the  SA  construct  by  dividing  it  into  three  hierarchical 
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levels.  Level  1,  the  lowest  level  of  SA,  involves  perceiving  cues  from  the  environment. 
Level  2  involves  integrating  these  perceptual  cues  with  other  information  and 
comprehending  their  meaning  and  significance.  Level  3,  the  highest  level  of  SA,  involves 
projecting  from  current  events  and  anticipating  future  events. 

2.2  Model 

Based  on  this  definition,  Endsley  [8]  developed  a  model  of  SA  in  dynamic  decision 
making.  A  diagram  of  Endsley's  model  is  shown  in  Figure  1. 


According  to  the  model,  SA  is  clearly  separated  from  decision  making  and  performance. 
Decision  making  and  performance  are  shown  as  separate  stages  that  will  proceed 
directly  from  SA.  A  high  level  of  SA  does  not  however  guarantee  high  quality  decision 
making  or  performance.  Several  major  factors  are  shown  to  influence  this  process.  All 
three  stages  of  the  process  can  be  influenced  by  individual  factors  such  as  innate 
abilities,  experience,  training,  fatigue,  propensity  for  risk  taking1,  and  task  factors  such 
as  workload,  stress,  and  complexity. 


2.3  SA  Measures 

As  technology  advances,  the  enhancement  of  operator  situation  awareness  has  become  a 
major  design  goal  for  those  developing  new  technologies  to  increase  performance.  The 
ability  to  objectively  evaluate  the  impact  of  the  technology  on  SA  is  therefore  essential. 
To  this  end  various  methods  have  been  developed  for  measuring  SA.  Several  of  these 
measures  which  are  relevant  to  infantry  SA  are  reviewed  in  the  following  section. 


1  The  propensity  for  risk-taking  is  expected  to  affect  only  the  latter  two  stages  of  the  process. 
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2.3.1  Objective  Measures  of  SA 

2.3.1. 1  Situation  Awareness  Global  Assessment  Technique 

SAGAT  [5]  is  a  direct  experimental  technique  for  measuring  SA  through  queries  or 
probes  and  provides  an  objective  assessment  of  SA  through  a  comparison  between  the 
real  and  perceived  situation.  The  SAGAT,  originally  developed  for  the  aviation  domain 
(commercial  pilot,  fighter  pilot,  air-traffic  control  e.g.  [9-11]),  has  also  been  applied, 
sometimes  in  a  modified  form,  in  many  other  domains  including  infantry  (section, 
platoon  and  brigade  level  in  real  and  virtual  environments  e.g.  [4, 12, 13]),  automotive 
(driving,  navigation  e.g.  [14-18]),  nuclear  energy  (nuclear  power  plant  operation,  tele¬ 
operations  of  nuclear  materials  handling  e.g.  [19-20  ],  and  medicine  [21]). 

The  SAGAT  consists  of  a  pool  of  questions  that  elicit  information  from  an  operator  in  a 
simulated  environment  across  all  three  levels  of  SA.  The  simulation  is  'frozen'  at 
random  while  the  operator  answers  a  series  of  randomly  selected  questions  about  the 
current  situation.  These  questions  correspond  to  the  operator's  SA  requirements. 
Temporary  freezes  in  the  simulation  must  be  of  a  short  duration  to  minimise 
intrusiveness  and  memory  decay.  Randomisation  of  questions  is  necessary  to  counter 
any  possible  learning  effects.  After  the  completion  of  the  simulation,  the  answers  are 
evaluated  on  the  basis  of  what  was  actually  happening  during  the  simulation. 

The  main  advantage  of  SAGAT  is  that  it  allows  an  objective,  unbiased  index  of  SA  [22]. 
The  SAGAT  also  measures  SA  directly  by  asking  subjects  for  information  pertaining  to 
the  operational  environment.  Another  strength  of  the  method  is  that  it  possesses  a  high 
degree  of  content  validity2  based  on  the  SA  requirements  analyses  used  to  create  the 
queries.  SAGAT  has  also  been  shown  to  have  criterion  validity3,  predicting  operator 
performance  in  an  air  combat  task  [24]. 

The  main  disadvantage  of  the  SAGAT  is  the  perceived  intrusiveness  of  freezes  in  the 
exercise/ simulation  to  collect  data,  and  the  criticism  that  the  technique  might  not 
provide  a  true  reflection  of  operator's  SA  as  it  relies  on  memory  [22]. 

2.3.1.2  Behavioural  Measures  for  Inferring  SA 

Infantry  leaders  and  soldiers  might  be  expected  to  act  in  certain  ways  based  on  their  SA. 
Therefore,  some  information  about  SA  may  be  inferred  from  examining  behaviours  on 
specific  subtasks  of  interest  [25].  Such  behavioural  indices  might  include  reaction  time 
to  make  a  response  (verbal/non-verbal),  time  to  complete  a  scenario,  loss  exchange 
ratio,  and  decision  making,  whereby  a  particular  conclusion  is  used  to  infer  the  SA  that 
underlies  the  decision  [25]. 

The  advantage  of  using  behavioural  indices  to  infer  SA,  is  that  the  measures  are 
objective,  readily  observable  and  usually  non-intrusive.  A  disadvantage  of  these 
measures  is  that  they  assume  what  appropriate  behaviour  should  be,  given  the  soldier's 
SA.  These  assumptions  may  not  always  be  warranted.  For  example,  different  soldiers 
may  respond  very  differently  to  the  same  situation  if  they  choose  different  combat 


2  Content  validity  refers  to  the  extent  to  which  a  measure  of  some  variable  samples  a  domain, 
such  as  a  field  of  knowledge  or  a  set  of  job  behaviours  [23]. 

3  Criterion  validity  refers  to  the  extent  to  which  a  test  predicts  performance  [23]. 
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procedures,  and  may  prioritise  tasks  differently  than  expected,  or  not  act  in  an  overt 
way,  even  though  they  recognise  and  understand  the  information  at  hand  [25]. 

2.3.1. 3  Performance  Measures  for  Inferring  SA 

While  behavioural  and  performance  measures  are  similar,  in  general,  performance 
measures  also  reflect  the  ability  of  the  soldier  to  carry  out  the  desired  action  (e.g. 
marksmanship)  as  well  as  his  choice  of  behaviours  (e.g.  shoot  at  the  target)  [25]. 

Like  behavioural  measures,  the  advantage  of  using  performance  measures  as  indicators 
of  SA  is  that  the  measures  are  objective,  observable  and  usually  non-intrusive.  Several 
limitations  exist  in  using  performance  data  to  infer  SA  however.  Global  performance 
measures  (e.g.  success  in  meeting  a  goal,  kills  in  a  battle)  suffer  from  problems  of 
diagnosticity  and  sensitivity.  Many  moderating  factors  can  influence  the  link  between 
SA  and  performance,  so  global  performance  measures  will  only  provide  an  indirect 
indication  of  SA.  Determining  appropriate  measures  for  discrete  task  performance  is 
also  difficult.  This  stems  from  the  interactive  nature  of  SA  sub-components.  For 
example,  a  new  system  or  training  technique  enhancing  SA  on  one  factor  may 
simultaneously  reduce  SA  on  another,  unmeasured  factor.  Unless  performance 
measures  are  chosen  carefully,  these  problems  may  not  be  detected.  In  addition,  it  is 
quite  easy  for  participants  to  bias  their  attention  to  a  single  issue  which  is  under 
evaluation  if  they  know  or  suspect  the  purpose  of  the  study  [25]. 


2.3.2  Subjective  Measures  of  SA 

2.3.2. 2  Situation  Awareness  Behaviourally  Anchored  Rating  Scale 

SABARS  [26]  is  a  direct  subjective  measure  of  SA  that  uses  expert  observations  to  rate  an 
operator  on  behaviours  linked  to  SA.  SABARS,  originally  developed  for  the  infantry 
domain,  measures  SA  at  the  platoon  level,  and  is  based  on  the  SA  requirements  analysis 
of  infantry  platoon  leaders. 

The  SABARS  metric  consists  of  27  behaviours  and  actions  linked  to  SA  in  Military 
Operations  in  Urban  Terrain  (MOUT)  missions.  During  virtual  or  field  simulated 
missions,  observers/controllers  observe  the  platoon  leaders  and  rate  performance  on  a 
five  point  scale  (very  poor  to  very  good)  at  the  end  of  each  mission. 

SABARS  has  several  advantages  for  rating  SA  in  the  field.  Firstly,  it  is  relatively 
unobtrusive  -  unlike  SAGAT  it  does  not  require  periodic  stopping  of  the  exercise. 
Secondly,  the  SABARS  items  describe  concrete  behaviours  that  are  readily  observable, 
thus  it  is  a  relatively  direct  and  objective  approach  to  assessing  SA.  Thirdly,  the  method 
is  viewed  as  relevant  and  acceptable  by  users.  For  example,  the  Royal  Norwegian  Army 
Academy  plans  to  incorporate  SABARS  into  its  after  action  review  procedures  [27], 

The  main  disadvantage  of  the  method  it  that  the  SABARS  items  are  mission  specific,  like 
SAGAT.  Therefore  a  SA  requirements  analysis  or  other  method  must  be  employed  to 
generate  SABARS  items  for  each  specific  type  of  mission  that  is  being  evaluated  [27]. 


4 


DSTO-TR-1628 


2. 3.2.2  Situation  Awareness  Rating  Technique 

SART  [28]  is  a  subjective  SA  measure  that  uses  operator  self-ratings  to  assess  perceived 
SA.  SART  assumes  that  operators  use  understanding  of  situations  in  making  decisions, 
that  this  understanding  is  consciously  available,  and  that  it  can  be  quantified  and  made 
unambiguous. 

Originally  developed  for  the  aviation  aircrew  domain,  SART  uses  a  self-rating 
questionnaire  to  measure  subjective  SA  in  3  or  10  dimensions.  The  3-dimensional  (3-D) 
SART  is  an  abbreviated  version  of  the  10-dimensional  (10-D)  SART  that  is  used  when 
the  10-D  version  would  be  too  intrusive  or  time  consuming  for  the  measured  task. 

The  SART  dimensions  reflect  generic  SA  constructs,  and  were  derived  from  an  extensive 
study  of  the  subjective  descriptions  of  43  different  air  warfare  scenarios  by  86  pilots. 
Because  the  constructs  are  general  in  nature  they  are  applicable  to  other  domains.  The  10 
generic  SA  constructs  are:  instability  of  situation,  variability  of  situation,  complexity  of 
situation,  arousal,  spare  mental  capacity,  concentration,  division  of  attention, 
information  quantity,  information  quality,  and  familiarity.  The  10  SA  constructs  are 
grouped  into  three  broad  domains  to  form  the  3-D  SART.  Attentional  demand 
encompasses  constructs  of  instability  of  situation,  variability  of  situation,  and 
complexity  of  situation;  attentional  supply  includes  the  constructs  of  arousal,  spare 
mental  capacity,  concentration,  and  division  of  attention;  and  understanding  incorporates 
information  quantity,  information  quality,  and  familiarity. 

SART  analysis  begins  with  the  creation  of  scenarios  or  stimulations  that  feature  the 
situations  of  interest.  During  these  trials  or  scenarios,  the  operator  is  provided  with 
either  a  3-D  or  10-D  SART  chart  to  record  his/her  perceptions  of  SA  at  a  given  point  in 
time.  When  using  the  3-D  SART,  the  subjects  typically  mark  their  rating  along  the  3 
dimensions  on  a  continuous  100-millimeter  line  with  the  endpoints  low  (0  mm)  to  high 
(100  mm).  Current  implementations  of  the  10-D  SART  employ  a  7-point  rating  scale 
from  low  to  high  for  each  of  the  10  SA  constructs,  along  which  the  subjects  quantify  their 
qualitative  observations.  The  scores  from  the  charts  are  then  statistically  analysed  to 
determine  how  aspects  of  the  task  affect  SA.  In  addition  to  scale  ratings  on  the  3-D 
version,  a  summative  score  is  derived  that  is  interpreted  as  an  estimate  of  overall  SA. 
This  score  is  derived  from  the  following  algorithm:  SA(calc)  =  Understanding  — 
(Demand  —  Supply).  This  formula  was  derived  from  theoretical  understandings  of  how 
the  three  domains  interact. 

An  advantage  of  SART  is  that  it  provides  a  certain  level  of  diagnosticity  [29],  Resolving 
SA  into  individual  SART  dimensions  provides  some  diagnostic  and  predictive 
indicators  for  delineating  the  strengths  and  weaknesses  associated  with  SA  as  measured 
by  the  scale. 

Proponents  suggest  that  workload  is  an  integral  part  of  the  SA  construct  and  that 
understanding  the  relationship  between  SA  and  workload  is  essential  in  order  for  SA  to 
be  an  effective  design  evaluation  measure  [28].  Thus,  an  advantage  of  SART  is  that  since 
it  takes  into  account  the  supply  and  demand  of  attentional  resources,  it  should  provide 
some  measure  of  how  changes  in  workload  affect  SA  [29], 

A  criticism  of  SART  is  that  including  workload  elements  within  the  SA  scale  confound 
the  measure  of  SA  by  confusing  it  with  workload  [29].  SART  has  been  shown  to  be 
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correlated  with  performance  measures,  but  it  is  uncertain  whether  this  is  due  to  the 
workload  component  or  understanding  (SA)  component  of  the  scale  [30]. 

Another  disadvantage  of  the  method  is  that  the  necessity  and  sufficiency  of  the  10 
general  constructs  within  the  3  domains  has  not  been  clearly  established,  and  the 
developers  acknowledge  that  considerable  scope  for  scale  development  remains  [29]. 

2.3.23  Mission  Awareness  Rating  Scale 

The  MARS  [30]  is  a  modified  version  of  the  Crew  Awareness  Rating  Scale  (CARS)  [31] 
adapted  to  the  military  infantry  domain. 

The  instrument  consists  of  two  subscales.  One  assesses  SA  content  (Content  Subscale) 
and  the  other  assesses  SA  workload  (Workload  Subscale).  Each  subscale  consists  of  four 
questions  that  address  the  three  levels  of  SA  as  defined  by  Endsley  [5]  -  identification, 
understanding,  prediction.  In  addition,  a  fourth  question  deals  with  how  well  mission 
goals  can  be  identified.  The  Content  Subscale  contains  questions  relating  to  the  subject's 
ability  to  identify,  comprehend,  predict  and  decide  in  the  given  mission.  The  Workload 
Subscale  contains  questions  relating  to  the  subject's  difficulty  in  identifying, 
comprehending,  predicting  and  deciding  in  the  given  mission. 

At  the  completion  of  a  virtual  or  field  simulated  mission,  participants  are  asked  to  self- 
assess  their  level  of  SA  and  the  amount  of  workload  required  to  achieve  that  level  of  SA 
along  a  scale  from  very  easy/ well/aware  to  very  difficult/ poor/ unaware. 

The  main  advantage  of  MARS  is  that  it  is  generic  and  does  not  need  to  be  tailored  to 
each  specific  mission  or  usage.  MARS  is  also  easy  for  respondents  to  understand  and 
use,  and  is  acceptable  in  field  training  exercises  due  to  its  unobtrusiveness  [27]. 

A  disadvantage  of  MARS  is  that  it  is  not  yet  clearly  established  that  the  instrument  in 
fact  measures  SA.  In  order  to  demonstrate  that  MARS  actually  measures  SA  per  se,  it  is 
necessary  to  compare  MARS  responses  against  a  more  objective  and  validated  measure 
e.g.  SAGAT  [5]. 

23.2.4  Post  Trial  Participant  Subjective  SA  Questionnaire 

The  PSAQ  [33]  is  a  three-item  instrument  designed  to  assess  a  subject's  perceived  level 
of  SA,  workload  and  quality  of  performance.  The  PSAQ  is  administered  at  the 
conclusion  of  a  scenario  and  each  item  is  rated  on  a  five-point  scale.  Subjects  are  asked 
to  self-rate  how  hard  they  were  working  (1  =  not  hard;  5  =  extremely  hard),  how  well 
they  performed  (1  =  extremely  poorly;  5  =  extremely  well),  and  how  aware  they  were  of 
the  evolving  situation  during  the  scenario  (1  =  not  aware;  5  =  completely  aware). 
Subjects  are  also  given  the  opportunity  to  write  down  any  further  'comments'  at  the  end 
of  each  item.  The  PSAQ  is  a  relatively  new  technique,  and  as  such  its  utility  and  validity 
has  not  yet  been  extensively  tested. 
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3.  Method 


3.1  Exercise  Silicon  Shrike 

3.1.1  Introduction 

The  study  consisted  of  a  free-play  exercise.  Free-play  exercises  realistically  replicate 
most  of  the  critical  aspects  of  a  soldier's  environment  and  are  a  standard  method  of 
training  individuals  and  units  in  fundamental  combat  skills.  In  each  trial,  ten  Air  Field 
Defence  Guards  participated  in  two  vignettes,  loosely  scripted  around  the  mission 
scenario  'Exercise  Silicon  Shrike'  [3].  In  the  enhanced  trial,  the  contribution  of  the  ISR 
(Motorola  GP300  Radios)  to  the  SA  of  the  individual  section  members  was  evaluated.  A 
diagram  of  the  communication  flow  during  baseline  and  enhanced  trials  is  shown  in 
Figure  2. 
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Figure  2:  Baseline  and  enhanced  section  communication  flow 
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3.1.2  Silicon  Shrike  Scenario 

The  Silicon  Shrike  scenario  involved  a  Ready  Reaction  Force  (RRF)  section  being  tasked 
to  respond  to  a  theoretical  enemy  -  Musorian  Armed  Forces  Special  Forces  (MAF  SF)  - 
activity  within  Tactical  Area  of  Responsibility  (TAOR)  Bulldog.  The  intent  of  the  MAF 
SF  was  to  resupply  and  regroup  before  conducting  further  operations  against  Australian 
Defence  and  Government  installations  and  personnel.  The  mission  of  the  RRF  section 
was  to  destroy  all  enemy  elements  within  TAOR  Bulldog  in  order  to  deny  their  resupply 
and  further  operations  against  Commonwealth  infrastructure  and  personnel. 

3.1.3  RRF  Section 

The  section  is  commanded  by  a  Section  Commander,  who  is  responsible  for  three 
smaller  manoeuvre  groups  within  the  Section;,  these  are  the  scout  group,  the  rifle  group 
and  the  assault  group.  The  scouts,  who  operate  forward  of  the  Section,  are  responsible 
for  gathering  information  and  reporting  signs  of  enemy  activity.  The  second-in- 
command  (2IC)  is  the  leader  of  the  gun  group.  This  group  carries  one  of  the  machine 
guns  in  the  Section  and  is  usually  used  to  provide  fire  support  during  a  Section  assault 
on  an  enemy  position.  The  assault  group  is  usually  employed  to  close  with  and  assault 
the  enemy  position.  For  the  purpose  of  the  current  study,  a  differentiation  is  made 
between  the  command  group,  namely  the  section  commander  and  2IC,  and  the  rest  of 
the  section.  The  command  group  has  a  higher  responsibility  for  the  whole  section.  This 
means  their  SA  requirements  are  more  extensive  than  those  of  the  other  section 
members,  as  has  been  reported  previously  [25]. 

3.1.4  Subjects 

Ten  Air  Field  Defence  Guards  from  1-Air  Field  Defence  Squadron  (AFDS)  participated 
in  the  experiment.  Subjects'  age  ranged  from  18  to  37  years  (M=23.31,  SD=  5.54).  The 
overall  military  experience  was  low,  with  time  in  service  ranging  from  10  months  to  5 
years  (Median  =  1).  In  both  baseline  and  enhanced  trial,  seven  section  members 
comprised  the  friendly  force  and  three  section  members  comprised  the  enemy  force. 
Two  military  subject  matter  experts  (SME)  participated  in  the  trial,  one  as  the  Platoon 
Commander  (CAPT)  for  the  RRF  section  as  well  as  an  observer  for  the  study,  and  one  as 
an  observer  (W02)  but  only  for  parts  of  the  activities 

3.1.5  Murray  Bridge  Range  Complex 

The  experiment  was  conducted  at  the  Murray  Bridge  Range  Complex  at  Murray  Bridge, 
South  Australia.  The  Area  of  Operations  (AO)  was  a  4  km2  site  north  east  of  the  Range 
Complex.  The  vegetation  consisted  mainly  of  gum  trees,  with  density  ranging  from 
scattered  to  medium  to  the  north,  and  medium  to  dense  to  the  south.  Prominent  features 
found  within  the  AO  included  enemy  village  (six  buildings).  Mobile  Mechanical  Range 
(MMR;  2  buildings  and  1  tower),  high  features  (76  m,  84  m),  quarry,  gates  (three),  razor 
wire  fence  and  unsealed  roads.  A  map  of  the  TAOR  Bulldog  is  shown  in  Figure  3. 
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Figure  3:  Murray  Bridge  Range  Complex-  TAOR  Bulldog 


3.1.6  Free-play  Vignettes 

Two  free-play  vignettes  appropriate  for  a  section  activity  were  used  in  each  trial.  The 
study  was  conducted  over  two  days.  The  activities  to  be  performed  were  basic  Infantry 
Minor  Tactics  (IMTs)  at  a  baseline  section  level.  At  this  level.  Air  Field  Defence  Guards 
are  suitable  subjects,  as  they  are  trained  to  perform  similar  activities  to  Army  infantry. 
This  was  confirmed  by  SME  observation  of  the  exercise.  The  first  vignette  was  a  RRF 
patrol.  The  patrol  distance  of  2.15  km  took  1  hr  48  min  to  complete  at  patrol  pace. 
During  the  baseline  trial  the  section  patrolled  from  the  MMR  to  the  Enemy  Village. 
During  the  enhanced  trial  the  section  patrolled  from  the  Enemy  Village  to  the  MMR.  The 
second  vignette  was  a  section  attack.  In  both  trials  the  section  attack  took  20  minutes  to 
complete,  and  occurred  approximately  500  m  south  from  the  end  location  of  the  Patrol. 
The  baseline  RRF  patrol  followed  by  baseline  section  attack  occurred  on  the  first  day.  On 
the  second  day  the  participants  undertook  the  enhanced  RRF  patrol  and,  subsequent  to 
that,  enhanced  section  attack. 

3.2.  SA  Question  Development 

A  crucial  part  of  the  DQT  is  the  development  of  the  pool  of  SA  questions.  During  the 
exercise  SA  questions  that  are  appropriate  to  the  situation  must  be  asked.  Since  SA 
covers  many  aspects,  only  a  subset  of  these  can  be  elicited  at  a  particular  time.  In  the 
current  study,  the  aim  was  to  include  as  many  SA  elements  as  possible  in  the  limited 
number  of  probing  sessions  available. 

SA  questions  are  developed  from  the  knowledge  of  the  information  required  by  the 
operators,  given  the  scenario,  mission  and  goals.  A  comprehensive  analysis,  which 
generally  requires  an  intensive  effort,  is  needed  to  elicit  the  information  requirements. 
For  SA  in  infantry  section  activities,  information  requirements  have  been  investigated  in 
detail  by  other  researchers  [1-2,  4, 12-13,  25-26,  31,  33-39].  It  reduced  considerably  the 
amount  of  analysis  required  for  the  present  study.  A  list  of  information  requirements 


9 


DSTO-TR-1628 


was  compiled  based  on  the  published  papers,  supported  by  military  publication  [6, 40- 
41].  This  list  was  divided  into  those  requirements  critical  for  an  RRF  Patrol  and  Section 
Attack  and  structured  around  the  military  SMEAC  [40]  (Situation,  Mission,  Execution, 
Administration  &  Logistics,  Command  &  Signals)  concept.  This  is  a  concept  used  by  the 
Army  when  preparing  orders.  These  information  requirements  were  then  converted  to 
question  form.  A  total  pool  of  68  questions  was  created  and  validated  by  three  SMEs 
with  extensive  military  experience.  (These  were  the  same  SMEs  as  those  mentioned  in 
Section  3.1.4,  aided  by  a  Lieutenant). 

The  validation  by  the  SMEs  focussed  on  three  aspects:  (i)  the  relevance  of  the  questions 
to  the  specific  scenario  and  mission  for  the  exercise,  (ii)  the  appropriateness  of  the 
questions  in  relation  to  the  roles  of  the  section  members,  (iii)  the  language  used  in  the 
questions  to  ensure  that  it  could  be  understood  by  the  potential  subjects. 

Due  to  the  need  to  minimise  the  intrusiveness  during  the  trials,  the  SMEs  were  asked  to 
identify  the  most  critical  questions  out  of  the  pool  so  as  to  limit  the  temporary  freezes  to 
five  minutes.  SMEs  were  also  asked  to  identify  to  whom  the  questions  were  relevant  (ie. 
Section  Commander,  Section  Member).  Following  this  process,  a  total  pool  of  46 
questions,  covering  all  three  levels  of  SA,  was  obtained.  Results  from  the  validation 
process  indicated  that  critical  SA  requirements  varied  according  to  the  level  of 
command.  Therefore,  whilst  the  number  of  questions  for  the  Section  Commander  and 
2IC  remained  at  46,  the  pool  of  SA  questions  for  the  section  members  was  reduced  to  33. 
(The  pool  of  questions  is  shown  in  Appendix  A). 

The  classification  of  the  questions  to  the  three  SA  levels  was  performed  by  the 
researchers  based  on  the  definition  of  SA  utilised  in  the  study.  Questions  relating  to  the 
perception  of  cues  were  classified  as  level  1,  as  were  those  that  elicited  information 
provided  to  the  subjects  in  the  exercise  brief,  for  example  "Outline  the  patrol  route  on 
the  map".  Questions  that  required  the  subjects  to  integrate  several  pieces  of  information 
to  assess  the  significance  of  cues  addressed  comprehension  and  were  classified  as  level 
2.  An  example  is  "Did  the  Patrol  help  you  discover  any  habits  of  the  MAF  SF?  If  so,  give 
details".  Finally,  level  3  questions  were  those  that  required  the  subjects  to  make  a 
prediction  of  future  events,  for  example  "Mark  on  the  map  the  most  likely  enemy  OP 
positions".  An  examination  of  the  questions  and  their  classification  as  shown  in 
Appendix  A  may  give  the  impression  that  the  classification  of  the  questions  to  the 
different  levels  is  arbitrary.  This  is  not  the  case,  as  it  depends  on  whether  the 
information  is  provided  to  the  subjects  or  whether  they  have  to  derive  it  from  perceptual 
cues  integrated  with  other  pieces  of  information.  For  example,  if  the  enemy  strength  is 
given  as  part  of  the  brief,  it  is  level  1.  If  it  is  concluded  from  observations  combined  with 
other  information,  it  is  level  2.  It  is  true  that  in  some  cases  the  delineation  into  the 
different  levels  is  not  clear-cut. 

Questions  were  then  allocated  to  the  baseline  and  enhanced  conditions,  albeit  ensuring 
that  all  three  levels  of  SA  were  represented.  In  addition,  various  questions  were 
allocated  to  the  different  phases  of  the  exercise  by  the  researchers  in  consultation  with 
the  SMEs.  These  were  based  on  the  intimate  knowledge  of  the  terrain,  what  the  subjects 
were  expected  to  do,  given  the  scenario,  mission,  and  cues  planted  to  indicate  "enemy" 
activities.  For  example  it  would  be  pointless  to  ask  the  question  "mark  on  the  map  any 
prominent  objects  and  reference  points"  on  the  phase  where  such  objects  did  not  exist. 
Not  all  of  the  questions  were  so  specific  in  nature,  in  which  case  they  could  be  asked  at 
different  phases. 
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A  summary  of  the  distribution  of  questions  is  shown  in  Table  1. 
Table  1:  Distribution  of  S A  questions 


Free-play  Vignette 

Number  of  SA  Questions 

Sect  Comd/ 
2IC 

Section 

Member 

RRF  patrol  -  Baseline 

13 

9 

RRF  patrol  -  Enhanced 

11 

8 

Section  attack  -  Baseline 

11 

8 

Section  attack  -  Enhanced 

11 

8 

The  SA  questions  consist  of  three  forms.  The  participants  are  required  to:  (i)  indicate 
locations  on  a  copy  of  a  map,  (ii)  sketch  a  diagram  showing  where  the  members  of  the 
section  are,  or  (iii)  simply  provide  written  answers  to  specific  questions.  The  questions 
do  not  include  those  that  require  either  a  "yes"  or  "no"  answer  or  those  of  multiple- 
choice  form. 

3.3  Data  Collection  Technique 

The  DQT  [4],  a  modified  version  of  the  SAGAT  was  used  to  provide  an  objective 
measure  of  SA.  The  DQT  differs  from  the  original  SAGAT  technique  in  that  it  is  applied 
in  a  live  exercise  rather  than  a  computer  simulation.  This  has  three  main  implications. 
Firstly,  all  the  questions  are  in  a  paper-based  rather  than  electronic  form.  Secondly,  the 
comparison  of  the  real  and  perceived  situation  is  performed  by  a  SME  rather  than 
comparing  subjects'  answers  to  data  collected  from  the  simulation  computers.  Thirdly, 
the  live  exercise  is  'frozen'  at  naturally  occurring  breaks  rather  than  in  a  totally  random 
fashion. 

For  the  current  study  the  DQT  was  piloted  in  activities  that  included  a  reconnaissance 
patrol  within  a  specific  scenario.  Six  Army  personnel  conducted  the  exercise  that  took 
place  on  the  eastern  area  of  DSTO  Edinburgh.  It  provided  the  opportunity  for  the 
researchers  to  refine  the  data  collection  technique  and  analysis.  The  subjects  involved  in 
the  pilot  study  did  not  participate  in  the  study  conducted  at  Murray  Bridge. 

Prior  to  the  start  of  the  activities  on  the  first  day  at  the  Murray  Bridge  Range  complex, 
the  subjects  were  provided  with  a  short  brief  about  the  aim  of  the  study  and  the  method 
by  which  the  data  was  going  to  be  collected.  They  were  also  asked  to  read  a  sheet 
containing  information  about  the  study  and  their  part  in  it,  following  which  they  were 
requested  to  sign  a  consent  form. 

During  both  baseline  and  enhanced  trials,  the  free-play  exercise  was  frozen  at  three 
naturally  occurring  breaks  in  the  normal  course  of  events,  each  of  five-minute  duration. 
At  the  start  of  the  break,  subjects  were  informed  that  they  had  to  answer  the  SA 
questions.  The  folders  containing  the  appropriate  sheets  were  distributed.  The  subjects 
sat  down  and  wrote  their  responses,  after  which  the  folders  were  collected.  Subjects 
answered  a  subset  of  the  pool  of  questions  about  the  current  situation,  based  on  the 
situation  awareness  requirements  of  the  soldier.  All  the  section  members  received  the 
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same  question  set.  The  Section  Commander  and  2IC  received  the  same  set  with  a  few 
additional  questions  relating  more  specifically  to  their  roles. 

After  the  completion  of  the  exercise,  subjects'  answers  were  evaluated  by  one  of  the 
SMEs  (CAPT)  on  the  basis  of  what  was  actually  happening  during  the  activities, 
supported  by  quantitative  Tactical  Engagement  Simulation  System  (TESS)  and  Global 
Positioning  System  (GPS)  data.  Questions  were  scored  with  1  for  a  completely  correct 
answer  and  0  for  an  incorrect  answer.  A  score  between  0  and  1  was  possible  if  the 
answer  was  only  partially  correct.  For  example  if  a  question  has  four  components,  0.25 
will  be  awarded  for  each  correct  response.  A  "don't  know"  or  nil  response  was  regarded 
as  a  totally  incorrect  answer.  Scores  were  then  summed  for  each  vignette  for  each 
person.  The  final  score  was  expressed  in  terms  of  the  total  correct  score  as  a  percentage 
of  the  total  maximum  possible  score. 

A  post-experiment  interview  with  the  two  military  SMEs  was  also  used  to  get  a 
qualitative  account  of  the  use  of  the  ISRs  during  the  enhanced  trial. 


4.  Results 


4.1.  Quantitative  Analysis 

Each  of  the  subjects'  responses  to  the  SA  questions  were  scored  by  the  SME  (CAPT)  in 
the  presence  of  the  researchers.  During  this  process,  three  questions  (Qll,  Q12,  Q35) 
were  removed  from  the  question  pool  as  they  were  found  to  be  irrelevant  for  the  specific 
activities. 

4.1.1  SA  scores 

The  SA  scores  are  shown  in  Figure  4.  It  shows  the  individual  scores  for  each  of  the  seven 
subjects  when  they  participated  in  the  four  activities,  namely  baseline  RRF  patrol, 
baseline  attack,  enhanced  RRF  patrol  and  enhanced  attack.  The  examination  of  the  data 
provided  some  insights  on  how  the  scores  changed  as  the  different  activities  were 
performed. 

Individual  analysis  revealed  that  Subject  4,  the  highest  ranked  (CPL)  and  most 
experienced  soldier  (5  years),  had  the  highest  total  correct  responses  summed  across  all 
vignettes  (M=0.89,  SD=0.27). 
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Figure  4:  Individual  SA  scores  across  the  four  activities  ( I -baseline  RRF  patrol,  2-baseline  attack, 
3-enhanced  RRF  patrol,  4-enhanced  attack) 

4.1.2  Paired  Samples  T-test 

Two  paired  samples  t-tests  were  computed  to  analyse  whether  there  was  a  significant 
difference  in  SA  between  baseline  and  enhanced  trials.  An  alpha  of  0.05  was  used  for  all 
statistical  tests.  Since  it  was  hypothesised  that  SA  may  increase  or  decrease  depending 
on  whether  the  ISR  was  used  correctly,  two-tailed  paired  samples  t-tests  were  used. 
Results  revealed  a  significant  increase  in  enhanced  RRF  patrol  (M=  0.55,  SD=0.22) 
compared  to  the  baseline  RRF  Patrol  (M=0.70,  SD=0.11):  t(6)=2.75  p=0.03.  The  rf  statistic 
of  0.56  indicated  a  large  effect  [42].  No  significant  differences  in  SA  were  found  between 
the  baseline  section  attack  (M=0.74,  SD=0.22)  and  enhanced  section  attack  (M=0.80, 
SD=0.22):  t(6)=0.64  p=0.55.  These  results  are  presented  graphically  in  Figure  5.  The  small 
amount  of  data  and  the  small  sample  size  did  not  permit  further  analysis  of  the  results 
in  terms  of  SA  level  1,  2  and  3. 

There  was  no  significant  difference  in  SA  between  the  baseline  and  enhanced  section 
attack. 
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Section  SA  Scores:  Baseline  vs  Enhanced 
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Figure  5:  Comparison  of  total  percentage  of  correct  responses  during  baseline  and  enhanced 
freeplay  vignettes.4 


4.2.  Qualitative  Analysis 

4.1.3  Post-experiment  interview 

A  debrief  with  the  two  SMEs  was  conducted  to  get  a  qualitative  account  of  the  actual 
use  of  the  ISR  during  the  baseline  and  enhanced  trials.  It  was  found  that  even  with  the 
availability  of  the  ISRs  in  the  enhanced  trials,  the  Section  Commander  relied  mainly  on 
hand-signals  to  communicate  with  the  section  members.  As  noted  by  the  SME,  the 
Section  Commander  was  very  disciplined  in  employing  the  radio. 

During  the  enhanced  patrol  activities,  the  ISRs  were  used  in  two  main  ways: 

•  Asa  substitution  for  handsignals  when  the  Section  Commander  wanted  to  pass 
on  a  simple  message  (e.g.  "move  in"). 

•  To  provide  instructions  during  more  complex  activities  e.g.  obstacle  crossings, 
building  clearances,  and  approaching  enemy  location. 

Overall  it  was  estimated  that  the  ISRs  were  used  approximately  35  times  in  the  patrol 
activities  -  approximately  one-third  during  obstacle  crossings,  one-half  during  enemy 
approaches  and  building  clearances,  and  one-quarter  for  simple  instructions.  For  all 
scenarios,  the  direction  of  the  communication  flow  was  from  the  Section  Commander  to 
the  rest  of  the  section,  except  during  obstacle  crossings  and  building  clearances  when 
section  members  sometimes  needed  to  initiate  communications. 

During  the  enhanced  attack  activities  the  ISRs  were  used  in  two  main  ways: 

•  To  provide  target  indications. 

•  To  enable  the  Section  Commander  to  get  an  initial  overview  of  the  situation. 


4  Although  the  SA  scores  for  the  Section  Attack  vignettes  appear  higher  overall  than  the  scores  for 
the  RRF  Patrol  vignettes,  the  pool  of  SA  questions  were  not  standardised  but  were  rather 
contextualised  to  the  particular  activities  to  be  performed.  Therefore  the  SA  scores  are  not 
comparable  between  vignettes. 
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During  the  enhanced  section  attack  the  radio  was  not  used  extensively.  During  the 
height  of  the  attack,  the  ISR  was  not  used,  for  three  main  reasons.  Firstly  the  noise  of  the 
firefight  made  hearing  the  radio  communications  very  difficult.  Secondly,  when 
communications  were  effected,  subjects  were  shouting  into  the  receiver  which  distorted 
the  transmission,  again  making  it  difficult  to  understand.  Thirdly,  shortly  after  the  initial 
communications  between  the  Section  Commander  and  the  Scout,  the  airwaves  became 
'clogged'  with  transmissions  from  many  different  section  members.  It  was  at  this  point 
that  the  Section  Commander  abandoned  the  use  of  the  radio  and  returned  to  verbal 
commands.  Towards  the  end  of  the  section  attack,  only  the  Gunner  and  Rifleman  had 
not  been  eliminated  in  exercise  play.  Dispersed  approximately  60  metres  from  each 
other,  the  ISR  was  used  several  more  times,  to  give  target  indications  and  navigational 
directions  so  that  they  could  locate  one  another. 


5.  Discussion 

5.1  Comparison  with  Previous  Research 

Two  previous  studies  [12,  35]  have  used  a  free-play  methodology  and  a  modified 
version  of  the  SAGAT  to  assess  the  impact  of  enhanced  technologies  on  SA  in  the 
infantry  domain. 

Redden  &  Blackwell  [35]  investigated  the  impact  of  MOUT  Advanced  Technology 
Concept  Demonstrator  (ATCD)  technologies  on  the  SA  of  a  joint  Army  and  Marine 
Company  using  a  free-play  methodology.  The  units  conducted  day  and  night  vignettes 
at  the  McKenna  MOUT  Site,  Fort  Benning,  under  baseline  (using  their  current 
equipment)  and  enhanced  (using  the  MOUT  ACTD  technology)  conditions.  A  freeze 
frame  technique  was  used  to  stop  the  exercise  at  three  phases:  before  the  attack; 
consolidation;  after  the  counter-attack  (post- vignette).  SA  was  assessed  using  a  modified 
version  of  the  SAGAT,  called  the  Questionnaire  Assessment  of  Knowledge  Technique. 
This  technique  was  shown  to  have  both  face5  and  content  validity. 

The  results  indicated  that  overall  the  technology  contributed  significantly  to  the  SA  of 
the  participants.  This  contribution  was  not  significant  during  the  daytime  but  was 
highly  significant  at  night.  The  technology  contributed  significantly  to  SA  in  the 
consolidation  phase  of  the  exercise .  During  this  phase,  the  troops  were  more  dispersed 
and  the  use  of  the  MOUT  ACTD  intra-squad  radio  appeared  to  increase  SA.  The  MOUT 
ACTD  did  not  significantly  impact  SA  during  the  attack  and  mission  completion  phases 
of  the  exercise. 

In  another  study  Redden  &  Blackwell  [12]  investigated  the  impact  of  the  intra-section 
radio  on  the  SA  of  a  Ranger  Squad.  The  Squads  conducted  day  and  night  vignettes  at 
the  McKenna  MOUT  site.  Fort  Benning  under  baseline  (no  intrasection  radio)  and 
enhanced  (intrasection  radio)  conditions.  The  vignettes  were  taken  from  a  Ranger 
Battalion  scenario.  SA  was  again  assessed  using  the  Questionnaire  Assessment  of 
Knowledge  Technique. 


5  Face  validity  refers  to  the  extent  to  which  a  measure  looks  as  though  it  measures  what  is 
intended  [23]. 
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The  results  showed  that  overall  the  intrasquad  radio  contributed  significantly  to  the  SA 
of  the  participants.  The  intrasquad  radio  did  not  have  a  significant  impact  on  SA  during 
the  day,  however  soldiers'  SA  was  significantly  better  at  night  with  the  radio.  A 
comparison  between  results  for  day  and  night  vignettes  revealed  a  significant 
degradation  in  SA  at  night  under  baseline  conditions.  There  was  no  significant 
difference  between  the  SA  scores  during  the  enhanced  day  and  night  vignettes 
indicating  that  the  use  of  the  intrasquad  radio  negated  the  SA  degradation  experienced 
at  night. 

A  direct  comparison  of  the  results  from  the  current  study  with  previous  research  [12, 35] 
was  not  possible  for  three  main  reasons.  Firstly,  the  current  study  was  conducted  in  a 
rural  environment  whilst  previous  studies  [12,  35]  were  conducted  in  a  MOUT 
environment.  Secondly,  a  direct  comparison  between  the  free-play  vignettes  was  not 
possible  as  the  vignettes  used  in  the  current  study  (RRF  Patrol;  Section  Attack)  were 
based  on  Tactics  Techniques  and  Procedures  (TTPs)  applicable  to  the  rural  environment, 
in  contrast  to  the  vignettes  used  in  previous  studies  which  were  based  around  TTPs 
applicable  to  MOUT  mission  scenarios  e.g.  building  clearances.  Thirdly,  the  majority  of 
subjects  used  in  the  current  study  (AFDGs,  ranked  Aircraftman  )  were  far  less 
experienced  than  the  subjects  used  in  previous  studies  (Platoon  Leader,  Platoon 
Sergeant,  Squad  Leaders  [35];  Rangers  [12]). 

Despite  this  it  is  worth  noting  that: 

•  The  non-significant  difference  in  SA  between  the  baseline  and  enhanced  section 
attack  in  the  current  study  is  consistent  with  Redden  &  Blackwell's  [35]  finding 
that  the  MOUT  ACTD  did  not  significantly  impact  SA  during  the  attack  phase  of 
the  exercise. 

•  The  increase  in  SA  in  the  enhanced  RRF  Patrol  in  the  current  study  is 
comparable  to  Redden  &  Blackwell's  [35]  finding  that  the  MOUT  ACTD 
technology  significantly  contributed  to  the  consolidation  phase  of  the  exercise 
when  troops  were  more  dispersed  and  the  intrasquad  radio  appeared  to  increase 
SA. 

•  The  increase  in  SA  in  the  enhanced  RRF  Patrol  in  the  current  study  is 
comparable  to  Redden  &  Blackwell's  [12]  finding  that  overall  SA  was  higher 
with  the  use  of  the  intrasquad  radio.  Although  the  radio  did  not  have  a 
significant  impact  on  SA  during  the  day,  Redden  &  Blackwell  [12]  note  that  the 
conditions  of  the  experiment  were  probably  the  toughest  conditions  for  showing 
the  contributions  of  the  radio.  Firstly,  the  MOUT  vignettes  required  less 
dispersion  than  several  other  tactical  situations  (e.g.  reconnaissance),  so  the 
soldiers  were  within  eye  contact  of  each  other  a  vast  majority  of  the  time. 
Secondly,  the  soldiers  used  in  the  experiment  (Rangers)  were  much  more 
experienced  in  MOUT  operations  than  typical  infantry  soldiers,  and  as  such 
were  more  highly  trained  in  the  use  of  non-verbal  communications  (e.g.  hand- 
signals),  which  reduced  the  need  for  the  radio  in  many  instances. 

Results  from  the  current  study  are  also  consistent  with  the  SME  observations  of  the 
exercise.  The  significant  increase  in  overall  group  SA  with  the  use  of  the  ISRs  during  the 
patrol  is  logical  as  even  though  SME  observation  revealed  that  the  ISR  was  not  used 
extensively,  there  was  still  a  greater  dissemination  of  information  throughout  the 
section.  This  was  because  the  absence  of  radio  communications  reflected  that  nothing 
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was  happening,  as  opposed  to  the  baseline  patrol  where  'silence'  may  have  reflected  not 
knowing  what  was  going  on. 

The  non-significant  difference  in  SA  between  the  baseline  and  enhanced  section  attack  is 
also  logical  given  that  SME  observation  of  the  section  attack  revealed  that  the  ISR  was 
hardly  used  during  this  activity. 

5.2  Validity  of  DQT 

The  DQT  and  free-play  methodology  was  able  to  successfully  discriminate  between 
baseline  and  enhanced  conditions  in  the  RRF  patrol  vignette,  despite  the  small  sample 
size  (n=7).  The  large  effect  size  indicated  that  the  impact  of  the  ISR  on  SA  was  worthy  of 
further  studies. 

5.2.1  Sensitivity 

The  range  of  total  correct  responses  obtained,  31%  to  81%  for  Patrol  vignettes;  39%  to 
100%  for  Attack  vignettes,  shows  that  the  DQT  has  good  sensitivity  as  it  is  able  to  reflect 
the  differences  in  subjects'  SA  over  a  large  range  of  scores. 

5.2.2  Face  Validity 

The  SME  validation  of  the  SA  question  set  and  observation  of  the  free-play  exercise  gave 
the  methodology  face  validity.  In  addition,  individual  analysis  revealed  that  the  most 
experienced  soldier.  Subject  4,  had  the  highest  total  correct  responses  summed  across  all 
vignettes  This  result  suggests  that  the  methodology  has  good  face  validity  as  it  is  able  to 
discriminate  between  experienced  and  novice  subjects. 

5.2.3  Content  Validity 

Although  a  formal  content  validity  analysis  was  not  conducted,  the  critical  information 
requirements  which  formed  the  basis  of  the  SA  questions  were  derived  from  a  literature 
review  of  scientific  and  military  publications  and  included  infantry  information 
requirements  lists  that  had  been  previously  validated  by  military  SMEs.  In  addition,  the 
SA  questions  used  in  this  study  were  validated  by  three  SMEs  (CAPT,  LT,  W02)  with 
extensive  military  experience. 

5.2.4  Construct  Validity 

A  strength  of  SAGAT  is  that  it  taps  directly  into  the  theoretical  SA  construct  of 
perception,  comprehension  and  projection,  and  therefore  has  a  high  construct  validity 
[22].  In  principle  the  DQT  emulates  SAGAT;  only  its  implementation  differs  slightly 
from  SAGAT.  Therefore  the  degree  of  construct  validity  of  the  DQT  is  expected  to  be 
comparable  to  that  of  SAGAT. 

Two  criticisms  have  been  levelled  at  SAGAT,  (i)  the  perceived  intrusiveness  of  freezes 
in  the  exercise/simulation  to  collect  data,  and  (ii)  the  degree  to  which  they  reflect 
memory. 
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5.2.4.1  Intrusiveness 

In  this  study,  an  attempt  was  made  to  minimise  intrusiveness  by  stopping  the  tree-play 
exercise  at  'natural  breaks'  of  a  short  5  minute  duration.  In  addition,  several  studies 
have  shown  that  a  temporary  freeze  in  a  simulation /exercise  to  collect  SAGAT  data 
does  not  impact  performance.  Endsley  [43]  conducted  two  studies  to  investigate  the 
impact  of  temporary  freezes  in  the  simulation  under  SAGAT  technique.  In  the  first 
study  she  investigated  how  long  after  a  freeze  in  the  simulation,  SA  information  could 
be  obtained.  In  the  second  study  she  investigated  whether  temporarily  freezing  the 
simulation  resulted  in  any  change  in  subject  behaviour.  Results  showed  that  freezes  (as 
many  as  3)  to  collect  SAGAT  data  did  not  have  a  significant  effect  on  subject 
performance,  and  that  subjects  were  able  to  report  their  SA  using  the  SAGAT  technique 
for  as  long  as  5  or  6  minutes  without  apparent  memory  decay.  Endsley  [24]  also 
investigated  whether  operator  performance  could  be  affected  by  the  mere  threat  of  a 
stop  to  collect  SAGAT  data.  That  is,  whether  operators  somehow  altered  their 
behaviour  during  trials  when  they  felt  that  they  may  be  stopped  and  tested  on  their  SA. 
Results  of  this  study  confirmed  previous  findings  and  showed  that  both  a  stop  or  even 
the  threat  of  a  stop  to  collect  SAGAT  data  did  not  have  a  significant  impact  on 
performance. 

It  should  be  noted  that  the  studies  conducted  by  Endsley  mentioned  above  took  place  in 
an  aviation  domain,  involving  pilots  in  high  tempo  operations.  In  the  current  study  in 
the  infantry  domain,  the  tempo  of  the  activities  was  much  lower.  Based  on  Endsley's 
findings,  it  is  unlikely  that  the  5  minute  breaks  in  the  infantry  exercise  would  have  had 
any  significant  impact  on  performance. 

5.2.42  Memory 

SA  measures  such  as  DQT  and  SAGAT  tap  into  memory  stores  (either  working  memory 
or  long-term  memory)  as  an  index  of  SA.  A  criticism  of  the  technique  surrounds 
whether  it  provides  a  good  representation  of  operator's  SA  or  whether  it  is  in  fact 
hindered  by  being  dependent  on  memory  [18]. 

It  has  been  widely  reported  [44]  that  retrospective  memory  for  past  cognitive  events  is 
poor,  and  so  a  technique  that  depends  on  retrospective  memory  would  clearly  represent 
a  problem  for  SA  measurement.  DQT  and  SAGAT,  however,  seek  to  probe  into  con¬ 
current  memory  by  placing  the  queries  immediately  following  the  event's  occurrence. 

Both  working  memory  and  long  term  memory  stores  are  important  for  SA.  Working 
memory  is  relevant  to  the  active  manipulation  and  use  of  information,  and  is  where 
dynamic  information  is  stored.  It  makes  sense  therefore,  that  the  contents  of  working 
memory  are  exactly  what  needs  to  be  tapped  by  an  SA  measure.  This  information  also 
needs  to  be  obtained  immediately  to  avoid  memory  decay. 

Long  term  memory  is  important  for  SA,  particularly  for  experienced  operators.  Endsley 
[10,  43]  found  that  experienced  pilots'  abilities  to  report  their  SA  via  SAGAT  was 
unaffected  by  how  long  after  the  freeze  the  questions  were  asked  (testing  intervals  from 
20  seconds  to  6  minutes).  The  current  study  appeared  to  support  Endsley's  finding,  that 
SA  information  was  available  for  quite  some  time  after  a  freeze  indicating  that  long-term 
memory  has  a  role  in  SA  as  well  as  working  memory. 
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An  explanation  for  this  result  was  that  the  experienced  expert  operators  had  long-term 
memory  stores  (such  as  schemata  and  mental  models)  that  served  to  organise 
information  and  have  an  effect  on  its  availability  for  a  measure  such  as  SAGAT. 

Another  concern  that  has  been  raised  is  whether  information  even  needs  to  be  in  the 
subject's  memory  (or  mental  representation)  in  the  first  place.  If  it  were  sufficient  that 
subjects  merely  reference  the  information  on  their  displays  when  needed,  SA  as  a 
dynamic  mental  representation  of  the  environment  would  not  be  needed.  However,  one 
of  the  most  frequent  causal  factors  for  SA  errors  involves  situations  where  all  of  the 
information  is  available  to  the  operator  but  it  is  not  attended  to  for  various  reasons  [45]. 
An  operator's  real  challenge  is  to  dynamically  locate  and  integrate  needed  information 
from  the  plenitude  available  in  the  environment.  For  this  reason,  SA,  as  a  measure  of  the 
operator's  ability  to  form  the  needed  understanding  of  the  dynamic  situation  and 
predict  future  events  in  the  short  term,  under  the  constraints  of  many  opposing 
demands,  is  the  measure  of  interest.  This  was  tested  in  the  current  study  through  SA 
level  2  and  level  3  questions. 

5.2A.3  Selective  Attention 

By  including  a  range  of  SA  questions  an  attempt  was  made  to  prevent  the  participants 
from  focussing  their  attention  on  a  narrow  range  of  factors  which  they  knew  they  would 
be  questioned  [22].  This  appears  to  have  been  successful  as  the  scores  did  not 
consistently  increase  with  time  as  the  trial  progressed  (See  Figure  4). 

5.2.4.4  Learning  Effects 

A  possible  learning  effect  was  that  the  patrol  route  for  the  baseline  trial  was  the  opposite 
to  that  for  the  enhanced  trial.  As  the  section  did  not  traverse  exactly  the  same  route  and 
covered  different  grounds,  the  effect  was  judged  to  be  minimal.  However,  learning 
effects  could  not  be  entirely  ruled  out.  If  they  were  present,  the  current  method  could 
not  discriminate  the  contribution  of  the  learning  effects  and  the  use  of  the  ISR. 

5.2.5  Limitation  of  the  study 

The  scarcity  of  resources  placed  a  limitation  on  the  design  of  the  study.  Only  one  group 
of  subjects  were  available  and  only  for  a  limited  time.  The  group  had  to  work  together 
as  an  infantry  section  and  could  not  be  split  into  smaller  entities.  Had  two  groups  been 
able  to  undertake  the  activities,  counterbalancing  the  use  of  the  ISR  would  have  been 
undertaken.  With  just  the  one  group,  little  would  have  been  gained  by  changing  the 
order  in  which  the  ISR  was  used.  In  addition,  only  one  primary  SME  was  available  for 
the  whole  study,  with  another  providing  assistance  from  time  to  time.  The  primary  SME 
was  therefore  involved  in  all  phases  of  the  study,  from  the  development  of  the  scenario 
and  SA  questions  to  the  scoring  of  the  responses  to  the  SA  probes.  Inter-rater  reliability 
test  was  not  performed  because  the  second  SME  was  not  available  to  observe  the  entire 
activities  and  therefore  did  not  know  the  correct  answers. 

The  reliability  of  the  method  was  not  tested  in  the  study,  but  SAGAT  has  been  shown  to 
have  a  high  reliability  in  studies  in  other  domains,  such  as  those  involving  fighter  pilots, 
nuclear  plant  operators  and  car  drivers  [22]. 
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6.  Conclusion 

6.1  Strengths  of  Method 

The  key  strength  of  the  DQT  method  is  that  it  assesses  SA  directly  by  asking  subjects  for 
information  pertaining  to  the  operational  environment.  Another  advantage  of  the  DQT 
method  is  that  it  is  relatively  inexpensive  because  no  special  equipment  is  needed  in 
order  to  apply  the  method. 

6.2  Weaknesses  of  Method 

A  limitation  of  the  DQT  method  is  that  intensive  effort  is  required  to  develop  the  SA 
questions,  which  have  to  be  contextualised  to  the  activities  to  be  performed  by  the 
subjects.  Another  disadvantage  of  the  method  is  that  it  is  often  difficult  to  determine  the 
'ground-truth'  in  free-play  exercises.  Individual  factors  such  as  the  lack  of  motivation  in 
answering  the  questions  may  also  affect  the  results  significantly. 

One  may  consider  employing  a  SABARS  type  technique  for  assessing  SA  in  which 
experts  observe  and  rate  operators'  behaviours  linked  to  SA  (see  Section  2.3.2.1).  It 
would  have  been  difficult  to  implement  SABARS  in  the  current  study.  The  nature  of  the 
terrain  and  activities  meant  that  the  section  was  spread  out  over  a  large  area  and  at 
times  some  section  members  were  obscured  by  vegetation.  An  observer  could 
reasonably  make  an  attempt  to  collect  data  from  one  or  two  subjects  simultaneously,  but 
not  more.  The  use  of  several  observers  might  well  be  distracting  to  the  participants  who 
are  required  to  move  in  a  covert  fashion. 

A  record  of  communication  between  the  members  of  the  section  may  be  expected  to 
provide  some  insights  on  the  group  processes,  such  as  co-ordination.  This  should  be 
considered  in  future  studies.  However,  even  such  a  record  will  not  provide  a  complete 
representation  of  communication  as  the  Section  members  still  use  hand-signals,  the 
tracking  of  which  is  difficult  to  accomplish. 

6.3  Future  Studies 

The  DQT  appears  to  be  a  valid  and  viable  methodology  that  can  be  used  in  the 
evaluation  of  the  impact  of  specific  SCS  technologies  on  SA.  Future  work  is  indicated 
below. 

6.3.1  Further  Validation  of  Method 

Further  validation  of  the  methodology  is  essential.  This  could  be  achieved  through: 

•  additional  trials  involving  a  higher  number  of  stops  thereby  allowing  more  SA 
probes  to  be  administered 

•  larger  sample  sizes 

•  Australian  Army  Infantry  sample  population 

•  audio  and  video  recordings  to  support  the  determination  of  'ground  truth' 

•  recording  of  radio  communication  when  ISR  is  used 

•  formal  training  in  the  ISR  operating  procedures  prior  to  commencement  of  field 
exercises 
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6.3.2  Comprehensive  Data  Analysis 

Due  to  the  small  sample  size  and  the  high  variability  of  the  data  set  the  assumptions  of 
many  non-parametric  statistical  tests  were  violated.  Thus  only  elementary  statistical 
analysis  was  possible.  Future  studies  with  a  larger  sample  size  could  avoid  many  of 
these  violations,  allowing  for  a  more  rigorous  statistical  analysis.  Where  parametric 
statistical  analysis  is  not  possible,  non-parametric  alternatives  should  be  considered. 

6.3.3  Tactics,  Techniques  and  Procedures 

In  the  selected  free-play  vignettes,  overall  the  ISR  was  not  used  extensively.  Therefore,  it 
may  be  beneficial  in  future  trials  to  select  TTPs  that  would  more  appropriately  show 
take-up  of  the  ISR  than  the  ones  used. 

6.3.4  Shared  SA 

As  well  as  examining  aggregated  group  SA  ,  it  would  also  be  beneficial  to  examine 
shared  SA  within  the  section.  The  members  of  an  infantry  section  do  not  work 
individually.  They  work  as  a  team  to  achieve  a  specific  mission.  Future  studies  should 
be  conducted  to  provide  an  understanding  of  team  SA  and  shared  SA  in  addition  to 
individual  SA. 
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Appendix  A:  SA  Question  Pool 


A.l.  RRF  Patrol  Questions 


Free-play 

Vignette 

SA 

Level 

Question 

Number 

Question 

1 

What  is  the  order  of  march? 

2 

Mark  on  the  map  your  location  at  the  start  of  the  Patrol. 

3 

Outline  the  patrol  route  on  the  map. 

4 

What  group  sizings  are  the  MAF  SF  operating  in? 

5 

What  are  the  suspected  current  activities  of  the  MAF  SF? 

6 

What  is  your  mission? 

7 

Mark  on  the  map  the  route  out  and  in. 

1 

8 

What  is  the  current  intent  of  the  MAF  SF? 

9 

What  signs  of  MAF  SF  activity  have  been  observed  so 
far?  (Baseline  Patrol) 

10 

What  signs  of  MAF  SF  activity  have  been  observed  so 
far?  (Enhanced  Patrol) 

11* 

Draw  a  rough  sketch  of  all  members  of  the  patrol. 

(Baseline) 

12* 

Draw  a  rough  sketch  of  all  members  of  the  patrol. 
(Enhanced) 

RRF 

13 

Mark  on  the  map  any  prominent  objects  and  reference 
points. 

Patrol 

14 

Mark  on  the  map  any  prominent  objects  and  reference 
points  observed  during  patrolling. 

15 

Were  there  any  obstacles/booby  traps?  If  so,  give 
details. 

2 

16 

Did  the  Patrol  help  you  discover  any  habits  of  the  MAF 

SF?  If  so,  give  details. 

17 

Where  are  you  currently  located  on  the  map?  (Baseline 
Patrol) 

18 

Where  are  you  currently  located  on  the  map?  (Enhanced 
Patrol) 

19 

What  are  the  strengths  and  weaknesses  of  the  MAF  SF? 

20 

How  likely  is  contact?  How  is  this  affecting  the 
movement  of  the  Patrol? 

21 

What  is  the  anticipated  reaction  of  the  MAF  SF  on 
discovery? 

3 

22 

Did  you  observe  any  likely  enemy  RV  Iocs,  enemy  FUPs 
or  enemy  hides?  If  so,  give  details. 

23 

Mark  on  the  map  the  most  likely  enemy  OP  positions. 

24 

What  are  the  possible  methods  of  infiltration  into  the 
TAOR  that  may  be  used  by  the  MAF  SF? 

*  These  questions  were  found  to  be  irrelevant  and  were  removed. 
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A.2.  Section  Attack  Questions 


Free-play 

Vignette 

SA 

Level 

Question 

Number 

Question 

25 

What  is  the  dress  of  the  MAF  SF? 

26 

Did  you  hear  the  target  indications  given?  If  so,  where 
were  they? 

27 

What  was  the  direction  of  contact  by  the  MAF  SF? 

1 

28 

What  is  the  order  of  succession  in  the  chain  of  command 
in  case  of  casualties? 

29 

What  weapons  did  the  MAF  SF  have? 

30 

Sketch  a  rough  diagram  of  the  distribution  of  all  section 
members  at  the  point  of  contact.  (Baseline  Attack) 

31 

Sketch  a  rough  diagram  of  the  distribution  of  all  section 
members  at  the  point  of  contact.  (Enhanced  Attack) 

32 

Who  was  on  your  left  and  right  during  the  assault? 

33 

How  many  casualties  does  the  enemy  have? 

34 

What  are  the  actions  on  contact  of  the  MAF  SF? 

Section 

35* 

What  is  the  section's  current  ammunition  status? 

Attack 

36 

How  many  casualties  do  you  have? 

2 

37 

How  did  you  maximise  cover  and  concealment? 

38 

At  what  point  did  you  feel  confident  of  the  enemy's 
disposition,  if  at  all? 

39 

What  course  of  action  did  you  take  (e.g.  close 
with/withdraw)  and  why? 

40 

How  likely  is  contact  with  the  MAF  SF?  Explain  why. 

41 

Where  are  the  most  likely  enemy  approaches  into  your 
TAOR? 

42 

What  is  the  most  likely  course  of  action  of  the  MAF  SF 
upon  contact? 

3 

43 

Where  is  the  most  likely  enemy  withdrawal  route? 
(Baseline  Attack) 

44 

Where  is  the  most  likely  enemy  withdrawal  route? 
(Enhanced  Attack) 

45 

Where  is  the  most  likely  enemy  rendezvous  point? 

46 

What  are  the  strengths  and  weaknesses  of  your  section? 

*  This  question  was  found  to  be  irrelevant  and  was  removed. 
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